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(57) Abstract 

A bridge for a multi-processor system includes bus biterfeccs for connection to an VO to of a first proces^ set, an I^ bus of 
a second processing set and a device bus. The bridge also inchides a memory subsystem and a bridge control rnedianisin. ™^^J^ 
control mechanism is operable to monitor operation of the first and second processing sets m a combined, lockstcp. opemtrn g »^)de and 
to be responsive to detection of a lockstep error to cause the bridge to be open^ 

die proci^ sets are buffered ma bridge buffer pcndmg resolution A respective buffer regwoBprovi^^ 

processing set. In an initial error mode, any complete device write accesses initiated by die processmg sets /^^f" 
Siffer. data is in transit tiJTOugh Ae teidge on entry to ti*e enw 

bridge control mediamsm is operable to permit nad access tt> the po^ 
to enable recovery from the error mode. 
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wo 99/66406 rCT,'US99/I2606 
TITLE: PROCESSOR biODGE WITH POSTED WRITE BUFF^ 

BACKGROUND OF THE INVEKnON 

5 This relates to feult tolerant con ip iita systems and to mechanisms for enabling recovety from an ennr 

during c^jeranon of the computer system. In particular, the invention relates to a processor bridge providing a 
posted write buffer for use in such a &uit tolerant computer system. 

In a fault tolerant computer system which provides detection of system errors, there can be some elapse of 
timg between detecting the cnor and taking an action to Kmit die imp.-xt of die enor, or to effect recovery from the 

10 error. During diis tinw, I/O cycles can stiU be pending widiin da system. This is because die processore and bus 
controllers in die system can inchidc I/O qierations wiach have afaeady been posted from die processors, and for 
^iiiich die jHocessors will no longer have any record On detecting an error, it b possible to identify a bus fauk to 
the issuing jrocessors or processor sets vMch would allow die processor to re-issue die I/O access following 
rcsohirion of die feuh. However, in die case of write accesses, die associated data would be lost if die accesses 

15 were siixq)ly bus crrored. 

Accordingly, die aim of die present invention is to provide a mechanism vAikh facilitates die taking of 
action to limit die impaa of an error or to completely recover from an error where pending I/O operations have 
already been initiated by a processor or processor set 

20 SUMMARY OF THE INVENTION 

Particular and preferred aspects of die invention are set out in die accGn^>anyiiig independent and 
dependent claims. Combinations of feanires from die dependent claims may be combined widi feamres of die 
independent claims as appropriate and not merely as explicitly set out in the claims. 

In accordance widi one aspect of die invention, there is provided a bridge for a multi-processor system. 
25 The bridge inchides a bus interfaces for connection to an 1/0 bus of a first processing set. an I/O bus of a second 
processing set and a device bus. The bridge also includes a memory subsystem and a bridge control 
mechanism. The bridge control mechanism is operable to monitor operation of the first and second processmg 
sets in a combined, lockstep, operating mode and to be responsive to detection of a lockstep error to cause die 
bridge to be operable in an error mode in which write accesses initiated by die processor sets are buffered in a 
30 bridge buffer pending resohition of the error mode. 

The provision of a bridge, including a bridge control mechanism widi a buffer for buffering wnte 
a c^^ c^g initiated by processor sets, enables write commands and the associated data which are located in 
various queues within die system to be stored pending resolution of an error. Accordingly, diis stored 
information can be used at least to limit die impact of die error and preferably completely to recover from diat 
35 error. It can further be used to identify die originator of the error. The buffer (posted write buffer) provides 
storage for I/O write accesses which occur after an error condition is detected. To allow die ac ces ses to be re- 
rraw die address to ^**ich die transaction would have occurred, any associated data, die type of write cuuuiund 
and also any byte enable information required to identify whidi parts of the data are valid, can be stored for 
later tise in the posted write buffer. 
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It should be noted ^ ' the bus interfaces refeitnced above need nr' Ve sqyarate componentr the 
bridge, bat may be mcoxporated in other con^Kmcnts of the bridge, and may indeed be simpW u)xinections or 
die lines of die buses concemcd. 

In an embodiment of the invention a respective buffer region is provided for each processing set 
5 The bridge control mrrharism of an embodiment of the invention is operable, in an initial error :node: 

to store in die posted write buffer any internal bridge write acces ses issued by the processing sets and 
to allow and to arbitrate any internal bridge read accesses initiatBd by the processing sets; and ' 

to store in a posted write buffer any coz^slete device write accesses initiated by the processing sets and 
to abort any device bus read accesses initiated by die processing sets. 
10 Moreover, die bridge control mechanism is operable in the initial error modc^ v/hm *n address part of 

a device write access has abeady been issued to the device bus, to buffer burst data parts of the device write 
access in one or more disconnect registers. 

The bridge control mechanism of an embodiment of die invention is c^Krable, in a primary error mode 
in v/idch a processing set asserts itself as a primary processing set 
15 to allow and to arbitrate any internal bridge write accesses initiated by die primary processing set, to 

discard any internal bridge write accesses initiated by any other processing set, and to allow and to arbitrate any 
internal bridge read accesses initiated by the processing sets; and 

to discard any device bus write accesses initiated by the processing sets and to abort any device bus 
read accesses initiated by die processing sets. 
20 A controllable routing matrix comiects the first processor bus interface, the second processor bus 

interface, the device bus interface and die memory sub-system. The bridge control mechanism is operable to 
control die routing matrix selectively to interconnect the first processor bus interface, the second processor bus 
interface, the device bus internee and the memory sub-system according to a current mode of operation. 

The bridge control mechanism of an embodiment of die invention further comprises an address decode 
25 mechanism coupled to the first processor bus interface, the second processor bus interface and the device bus 
interface, the address decode mechanism being operable to determine target addresses f^ write and read 
accesses. The initiator of die write and read accesses wiU be determiried by an arbitration mechanism. Ixutiator 
and target controllers control die routing of a path from an input to an output of die routing matrix, the initiator 
and target controllers being responsive to die arbitration address decode mechanisms, respectively. 
30 Ibe bridge control mechanism of an embodiment of the invention conqnises a comparator connected 

to die first and second processor bus interfaces. The con:q)arator is operable in die combined mode to detect 
differences between signals on die I/O buses of die first and second processing sets as indicative of a lockstep 
error. A bridge controller is connected to an output of the comparator, the bridge controller being operable, in 
response to a signal indicative of a lodcstep error output firom the comparator, to cause the bridge to cease 
35 operation in die combined mode and instead to operate in the error nnode. The master and target controllers are 
responsive to an output of die bridge controller to modify the path fiom the iiqnxt to the output of the routing 
matrix. 

In an embodiment of the invention a posted write buffer is configured in random accfs^ memory 
connected to or fanning part of the bridge. Disconnect registers are configured as separate hardware registers. 
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The bridge could u*;huk more than two jOTOSsw biK 
ofluitfa^ pr o cess ing sets. 

In accordance with another aspect of the invention, there is provided a multi-processor system 
comprising a first processing set having an yO ^ 
5 a bridge as set out above. Each processing set can inchide at least one processor, memory and one or more 

processing set I/O bus controUcts. 

In accordance wi4 a ftoher aspect of the invention, there is pnm^ 

processor system as set out above, the method comprising: 

monitoring operation of the first and second processing sets in a combined, lockstcp, operating mode; 

10 and 

on detection of a lockstcp error, buffering write accesses initiated by the processor sets in a buffer 
pcttdii^ resolution of the error. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Exemplary embodiments of the present invention will be described hereinafter, by way of example only, 
wiihreferenceto the accompanying drawings in which like referee 

Figure 1 is a schematic overview of a feuh tolerant computer system incorporating an embodiment of the 



invention; 



Figure 2 is a schematic overview of a specific implementation of a system based on that of Figure 1; 
20 Figure 3 is a schematic representation of one iirqjlernentation of a processing set; 

Figure 4 is a schematic representation of another example of a j^ocessing set; 
Figure 5 is a scheinatic rcpreseiiiation of a further processing set; 

Figure 6 is a schematic block diagram of an embodiment of a bridge for ihe system of Figure 1; 

Figure 7 is a schematic block diagram of storage for &e bridge of Figure 6; 
25 Figure 8 is a schematic block diagram of control logic of the bridge of Figure 6; 

Figure 9 is a schematic representation of a routing matrix of the bridge of Figure 6; 

Figure 10 is an example implementation of the bridge of Figure 6; 

Figure 11 is a state diagram ilhistrating operational states ofthc bridge of Figure 6; 

Figure 12 is a flow diagram illustrating stages in Ac operation of the bridge of Figure 6; 
30 Figure 13 is a detail ofa stage ofoperationfiom Figure 12; 

Figure 14 ilhistrates d» posting of UO cycles in the system of Figure 1; 

Figure 15 illustrates d» data stored in a posted write buffer; 

Figure 16 is a schematic representation ofa slot response register; 

Figure 17 illustrates a dissimilar data write stage; 
35 Figure 18 illustrates a modification to Figure 17; 

Hgure 19 illustrates a dissimilar data read stage; 

Figure 20 ilhistrates an alternative dissimilar data read stage; 

Figure 21 is a flow diagram summarismg Reoperation of a dissimilar data write mechanism; 
Figure 22 is a ^fr^^'^^'g' block diagram explaining arbitration within ihe system of Figure 1 ; 
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Figure 23 Bass^'^ db^^^illiutratmgtheoperatioiiofac^^ 

Ftgaxc is a state dr ^am iDussaisog the opciaiion of a fandgc attntet; 

Figure 25 is a timixi^, diagram for PQ ^gnai*; 

Figure 26 is a scb jxnatic diagram ilhistrating the opaatioa of the bridge of Figure 6 for direct memory 

F^ure 27 is a flow diagram iUustratiDg a dirtct memory access mr^sod in die bridge of Figure 6; and 
Figure 28 is a flow diagram of a re-itiiegration process jfyhiding &e monisoring of a diry RAM. * 

DESCRgTION OF THE PREFERRED EMBODIMENTS 

10 Figur; j is a schematic overview of a fault tolerant, confuting system 10 comprising a plurality of 

CFUsets (processing sets) 14 and 16 and a bridge 12. As shown in Figure 1, diere are two processing sets 14 
and 16, althcnigh in other embodiments diere may be three or more processing sets. The bridge 12 forms an 
intoface between the processing sets and I/O devices such as devices 28, 29, 30, 31 and 32. In this docmnent, 
the term ^'processing set^ is used to denote a groi^ of one or more processors, possibly including memory, 

15 which output and receive conmion outputs and inputs. It should be noted that die alternative term mentioned 
above, 'OUset", could be used iristrad, and dxat these tenzKS could be used interchangeably throughout this 
document Also, it should be tK>ted that &e term "bridge** is used to deix>te any device, apparatus or 
arrangement suitable for interconnecting two or more buses of the same ox different types. 

Tbe first processing set 14 is cozmected to the bridge 12 via a first processing set I/O bus (PA bus) 24, 

20 in die present instance a Peripheral Component Intcrcoimect (PCI) bus. The second processing set 16 is 
cormected to the bridge 12 via a second processing set I/O bus (PB bus) 26 of the same type as the PA bus 24 
(Le. here a PCI bus). The I/O devices are coimected to the bridge 12 via a device I/O bus (D bus) 22, in the 
present instance also a PCI bus. 

Ald)ough, in fiie particular exan^le described, the buses 22, 24 and 26 are all PCI buses, this is merely 

25 by way of example, and in other embodiments other bus protocols may be used and the D-bus 22 may have a 
different protocol fiom that of the PA bus and ibe PB bus (P buses) 24 and 26. 

The processing sets 14 and 16 and the bridge 12 are operable in synchronism under the control of a 
common clock 20, which is connected thereto by clock signal lines 21. 

Some of the devices including an Ethernet (E-NET) interface 28 and a Small Computer System 

30 Intcr&ce (SCSI) interface 29 are permanently coimected to the device bus 22, but other I/O devices such as I/O 
devices 30, 31 and 32 can be hot insertable into individual switched slots 33, 34 and 35. Dynamic field effect 
transistor (FET) switching can be provided for the slots 33, 34 and 35 to enable hot insertability of the devices 
such as devices 30, 31 and 32. The provision of the FETs enables an increase in its lengdi of die D bus 22 as 
only those devices ^^ch are active are switcbed on, reducing the effective total bus lengdt It will be 

35 a|)preciated that the number of I/O devices which may be coimected to the D bus 22, and die number of slots 
provided for &em, can be adjusted according to a particular implementation in accordance with specific design 
recpuremcnts. 

Figure 2 is a schematic overview of a particular iiiq>lementadon of a fault tolerant con^uter employing 
a bridge structure of the type illustrated in Figure 1. In Figure 2, the fauh tolerant conqniter system includes a 
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phnalhy (here four) of bridges 12 on first and star^ I/O mothcrl oaids (Mb 40 and MB 42) onto to increase 
tbe rnnnber of VO devices which may be connected and also to Jiwroyerel^ Thm. in 

the ernbodiment shown in Figure 2, two processing sets Uand .6aie each proWd^ 
set board 44 aDd 46. with the processing set boanls 44 and 46 'biidginsVtheyO motherboards MB 
42. A first, rnastcr clock source 20A is mounted on the first riwdierboaid 40 and a second, slaved 
20B is mouated on the second rrwtherboard 42. Qock signals arc supplied to the processing set boards 44 and 
46 via respective comections (not shown in Figure 2).- 

Fiist and second bridges 12.1 and 12.2 are mounted on the first I/O motherboard 40. The first bridge 
IZl is comiected to the processmg sets 14 and 16 P buses 24.1 and 26.1. respectively. Simibrly, the second 
bridge 12.2 is connected to the processing sets 14 and 16 by P buses 24 J and 26X respectively. The bridge 
12.1 B corrnected to an yO databns (D bus) 22.1 and the bridge 122 is connected to an yO datt^ 

22i 

Third and fourth bridges 123 and 12.4 are mounted on the second VO motherboard 42. The bridge 
\23 is comiected to dre processing sets 14 and 16 by P buses 24J and 26J. respectively. Similarly, the bridge 
4 is connected to die processing sets 14 and 16 by P buses 24.4 and 26.4. respectively. The bridge 123 is 
connected to an VO daabus (D bus) 223 and die bridge 12.4 is connected to an VO databus (D bos) 22.4. 

It can be seen diat die arrangement shown in Figure 2 can enable a large number of VO devices to be 
comiected to die two processmg sets 14 and 16 via die D buses 22.1. 22J, 22 J and 22.4 for eittier in^ 
die range of yO devices available, drproviding a higher degree of redundancy, or boflL 

Figure 3 is a schematic overview of one possible configuration of a processing set, such as die 
processing set 14 of Figure 1. The processing set 16 could have die same configuration. In Figure 3, a phiraUty 
of processors (here four) 52 are connected by one or more buses 54 to a processing set bus controUer 50. As 
shown in Figure 3, one or more processing set output buses 24 are connected to die processing set bus controller 
50, each processing set output bus 24 being comiected to a respective bridge 12. For example, in die 
armngement of Figure i. only one processing set yO bus (P bus) 24 would be provided, whereas in die 
arrangement of Figure 2, four such processing set yO buses (P buses) 24 would be provided. In die processing 
set 14 shown m Figure 3, individual processors operate usmg die common memory 56. and receive inputs and 
provide outputs on the conmton P bus(es) 24. 

Figure 4 is an alternative configuration of a processing set, such as die processing set 14 of Figure 1. 
Here a plurality of processortoemory groups 61 are connected to a common internal bus 64. Each 
processor/memory group 61 inchides one or more processors 62 and associated memory 66 connected to a 
internal group bus 63. An interface 65 connects die internal group bus 63 to die coirmion internal bus 64. 
Accordingly, m die arrangement shown in Figure 4. mdividual processmg groups, wid» each of die processors 
62 and associated inemory 66 are connected via a common internal bus 64 to a processing set bus conirolte 

35 The interfi«*s 65 enable a processor 62 of one processing group to operate not only on die data in its local 
memory 66, but also in die memory of anodier processing group 61 wifliin d» processing set 14. The 
processing set bus controller 60 provides a common interiiace between die common internal bus 64 and die 
processing set VO bus(es) (P bus(es)) 24 comiected to die bridge(s) 12. It should be noted diat ahhough only 
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two processing gnnips 61 are >wd in Figure 4, it wiU be appreciated diat sac* suurri^ is ziot iimitcd to this 
mnnter of processing groi^ 

Figure 5 iUastrates an ahcinative configuraticm of a | i n : May i ng set such as ilie j'/scessing set 14 of 
Figure 1. Here a sinq>le processmg set includes a single processor 72 and associated noenaiy 76 connected via 
5 a comnxm bus 74 to a processing set bus controller 70. The processing set bos coiitroBer 70 piovides an 
interface between die internal bus 74 and the processing set I/O bus(es) (F bus(e3)) 24 for connection to die 
bridge(s)lZ 

Aocordmgly, it will be ^spreciatcd from Figures 3, 4 and 5 diat the pro cessin g set may have many 
different fonns and that die particular choice of a particular processing set stricture can be ™^ on the basis of 

10 the processing requirement of a particular application and the degree of redun.^ancy required. In the following 
description, it is assumed that die processing sets 14 and 16 referred to have a structure as shown in Figure 3« 
although it will be appreciated that another form of processing set could be provided. 

The bridge(s) 12 are operable in a number of operating modes. These modes of operation wiU be 
described in mac detail later. However, to assist in a general understanding of the su uc t ui e of the bridge, die 

15 two operating modes wiD be briefly summarized here. In a first, combined mode, a bridge 12 is operable to 
route addresses and data between the processing sets 14 and 16 (via die PA and PB buses 24 and 26, 
respectively) and die devices (via die D bus 22). In this conibined mode, I/O cycles generated by the procesising 
sets 14 and 16 are con^saicd to ensure that both processing sets are operating correctly. Comparison failures 
force the bridge 12 into an error limiting mode (EState) m which device I/O is prevented and diagnostic 

20 information is collected. In the second, split mode, the bridge 12 routes and arbitrates addresses and data from 
one of the processing sets 14 and 16 onto the D bus 22 and/or onto the other one of the processing sets 16 and 
14, respectively. In this mode of operation, the processing sets 14 and 16 are not synchronized and no I/O 
conq>arisons are made. DMA operations are also permitted in bodi modes. As mentioned above, die different 
modes of operation, including the combined and split modes, will be described in more detail later. However, 

25 there now foUows a description of the basic structure of an example of the bridge 1 2. 

Figure 6 is a schematic functional overview of the bridge 12 of Figure 1. First and second processing 
set I/O bus interfaces, PA bus interface 84 and PB bus interface 86, are cormected to die PA and PB buses 24 
. and 26, reqxctively. A device I/O bus interface, D bus interface 82, is connected to the D bus 22. It should be 
noted that die PA, PB and D bus interfaces need not be configured as separate elements but could be 

30 incorporated in other elements of the bridge. Accordingly, within die conteact of diis doctiment, where a 
references is made to a bus interface, this does not require the presence of a specific s^»arate con^nent, but 
rather the capability of die bridge to connect to die bus concerned, for example by means of i^iysical or logical 
bridge connections for die lines of die buses concerned. / 

Routing (hereinafter termed a routing matrix) 80 is coimected via a first internal padi 94 to the PA bus 

35 interface 84 and via a second mtcmal padi 96 to die PB bus inter&ce 86. The routing matrix 80 is furdier 
connected via a diird internal padi 92 to die D bus interface 82. The routing matrix 80 is therdyy able to provide 
I/O bus transaction routing in bodi directions between die PA and PB bus interfaces 84 and 86. It is also able to 
provide routing in both directions between one or both of die PA and PB bus interfaces and die D bus interface 
82. The routing matrix 80 is coimected via a further internal path 100 to storage control logic 90. The storage 
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contTol logic 90 controls access to bridge icgistos 110 and to a random access memoiy (SRAM) 126. The 
ronting matrix 80 is therefore also operable to provide routing m both d!^^ 

imnfaces 84. 86 and 82 and the storage control logic 90. The nwting matrix 80 is controlled by bridge control 
logic 88 over control paths 98 and 99. The bridge control logic 88 is reqwnsive to control signals, data and 
5 addresses on internal paths 93, 95 and 97, and also to clock signals on flic cIockline(s) 21. 

In fljc embodiment of the invention, each of die P buses (PA bus 24 and PB bos 26) operates nnder a 

pa protocol Ihe processing set bus controlleis 50 (see Figure 3) also operate under the PO laotocoL- 
Accoidmgly. the PA and PB bus interfaces 84 and 86 each provide all die functionality required f« a 
compatible interfece providing both master and slave (^)eration for data transferred to and from flie D bus 22 or 
10 internal memories and registers of the bridge in the storage subsystem 90. Ihe bus intcrfeces 84 and 86 can 
provide diagnosdc information to internal teidge status registcre in the storage subsystem 90 on transition of the 
bridge to an error state (Estate) or on detection of an 1/0 error. 

The device bus interface 82 performs aD die functionality required for a PCI conq)liant master and 
slave interface for transferring data to and from one of die PA and PB buses 84and86. 11ieDbus82is 
15 operable during direct memory access pMA) transfers to provide diagnostic information to internal stams 
• registers in die storage subsystem 90 of the bridge on transition to an EState or on detection of an I/O error. 

F«ure 7 illustrates in more detafl dre bridge registers 1 10 and Are SRAM 124. The storage control 
logic 110 is connected via a pad! (e.g. a bus) 112 toa number of registercomponenis 114, 116, 118, 120. The 
storage control logic is also connected via a padi (e.g. a bus) 128 to the SRAM 126 in which a posted write 
20 buffer componem 122 and a dirty RAM component 124 are rnapped Although a particular configuratiOT 

components 114. 116. 118. 120. 122 and 124 is shown in Figure 7, diesc components may be configured in 
odier ways, widi odier conqxjnents defined as regions of a connnon memory (e.g. a random access memory 
such as die SRAM 126. widi die path 112/128 being formed by die internal addressing of fee regions of 
memory). As shown m Figure 7. die posted write buffer 122 and die dirty RAM 124 are mapped to different 
25 regions of die SRAM memoiy 126. whereas die registers 1 14. 1 16. 1 18 and 120 are configured as separate from 
the SRAM memory. 

Control and status registers (CSRs) 114 fomi imernal registers which allow die control of various 
operating modes of die bridge, allow die capmre of diagnostic information for an EState and for I/O errors, and 
control processing set access to PQ slots and devices connected to die D bus 22. These registers are set by 
30 signab from die routing matrix 80. 

Dissimilar data registers (DDRs) 116 provide locations for containing dissimilar data for different 
procesOTg sets to enable non-deterministic data events to be handled. These registers are set by signals from 

die PA and PB buses. 

Bridge decode logic enables a common write to disable a data comparator and allow writes to two 
" > 35 DDRs 116, erne for each processing set 14 and 16. . 

A selected one of die DDRs can dicn be read in-sync by die processing sets 14 and 16. The DDRs dnis 
provide a mcchaniin enabling a location to be reflected from one processing set (14/16) to anodier (16/14). 

Slot response registers (SRRs) 118 determine ownership of device slots on die D bus 22 and to aHow 
DMA to be routed to die appropriate processing set(s). These registers are linked to address decode logic 
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Disconnect regist ers ' ire used for the storage of data phases of an J ^le which is aborted whfle . 
* daiaisinlhedridgeonlhewaytDanodierbtK. The disconnect registexs 120 receive all data qoeoed in the bridge 
when a tairet device di sc o nne cts a transaction, or as the EState is dctcctrd Utese i c t^iMas are ccmnected to the 
routing matrix 80. The routing matrix can queue up to three data words ai^ byte enables. Provided dse initial 
5 addresses are voted as being equal, address target controDeis derive addresses whidi iuur -ii n i t as data is 
exchanged between die bridge and the destination (a target). Where a writer (for cxznapk a pr oce ssor I/O write, or 
a DVMA (D bus to P bus access)) is writing data to a target, this data can be caught in the bridge v/ben an error* 
occurs. Accordingly, fhis data is stored in die disconnect registers 120 whm an error occun. These disconnect 
registers can then be accessed on recovery from an £Statc to recover die data associated widi die write or read cycle 
10 which was in progress when the Estate was znrtiated. 

Although shown sq>arately, the DDKs 1 16» the SRRs 1 18 and die disconnect registers may form an 
integral part of die CSRs 114. 

Estate and error CSRs 1 14 provided for the capture of a failing cycle on die P buses 24 and 26, with an 
indicadon of die failing datunL Following a move to an EState, all of die writes initiated to the P buses are 
IS logged in die posted write buffer 122. These may be odicr writes that have been posted in die processing set 
bus controllers 50, or \^ch may be iiutiated by software before an EState interrupt causes the processors tq 
stop carrying out writes to the P buses 24 and 26. 

A dirty RAM 124 is used to indicate whidi pages of die main memory 56 of the p roce ssii ^ sets 14 and 
1 6 have been modified by direct memory access (DMA) transactions from one or nxire devices on the D bos 22. 
20 Each page (e.g. each 8K page) is marked by a single bit in die dirty RAM 124 which is set when a DMA write 
occurs and can be cleared by a read and clear cycle initiated on the dirty RAM 124 by a processor 52 of a 
processing set 14 and 16. 

The dirty RAM 124 and the posted write buffer 1 18 may bodi be mapped into die memory 124 in the 
bridge 12. This memory space can be accessed daring normal read and write cycles for testing purposes. 
25 Figure 8 is a schematic functional overview of the bridge control logic 88 shown in Figure 6. 

All of the devices coimected to the D bus 22 are addressed geographically. Accordingly, the bridge 
carries out decoding necessary to enable the isolating FETs for each slot before an access to those slots is 
initiated. 

The address decoding performed by the address decode logic 136 and 138 essentially permits four 
30 basic access types: 

- an out-of-sync access (le. not in the combined mode) by one processing set (eg. processing set 14 of 
Figure 1) to dttodier processing set (e.g. processing set 16ofFigure 1), in which case the access is routed from 
die PA bus interface 84 to the PB bus interface 86; 

- an access by one of the processing sets 14 and 16 in the split mode, or both processing sets 14 and 16 
35 in the combined mode to an I/O device on die D bus 22, in which case die access is routed via die D bus 

inter&ce82; 

- a DMA access by a device on die D bus 22 to one or bodi of the processing sets 14 and 16, which 
would be directed to bodi processing sets 14 and 16 in the combined mode, or to the relevant proccssmg set 14 
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or 16 if out-of-sync ana » in 9 spm mode 10 processing set 14 or 16 «iuv«i owns a slot in which dxe device is 
located;and 

- a PCI configuration access to de^ ices m I/O slots. 

As mentioned above, geogr^hic addressing is en^loycd Thus, for exarr^k, stot 0 on motherboard A 
5 has tiie same address when referred to by processing set 1 4 or by processins set 1 6. 

Geographic addressing is used in combination with the PQ slot FET switching. During a 
eonfiguration access mentioned above, separate device select signals arc provided for devices ^ch are not 
FET isobtcd. A single device select signal can be provided for the switched PCI slots as die FET signals can be 
used to enable a correct card. Separate FET switch lines are provided to each slot for separately switching die 
10 FETs for the slots. 

The SRRs 118, which could be incorporated in die CSR registers 114. are associated with die address 
decode functions. The SRRs 118 serve in a number of difFcrcni roles which will be described in more detail 
later. However, some of the roles are summarized here. 

In a combined mode, each slot may be disabled so that writes are sin^^ly acknowledged without any 
15 transaction occurring on die device bus 22, whereby the data is lost Reads will return meaningless data, once 

again without causing a transaction on the device board. 

In the split mode, each slot can be in one of three states. The states are: 

-Notowi^ 

- Owned by processing set A 14; 
20 - Owned by processing set B 16. 

A slot djat is not owned by a processing set 1 4 or 1 6 making an access (dus inchides not owned or un- 
owned slots) cannot be accessed. Accordingly, such an access is aborted. 

When a processmg set 14 or 16 is powered off, all slots owned by it move to die un-owned state. A 
processing set 14 or 16 can only clann an un-owned slot, it cannot wrest ownership away from anodier 
25 processing set This can only be done by powering off die other processing set, or by getting die odier 
processing set to relinquish ownership. 

The ownership bits are assessable and sellable while in the combined mode, but have no effect until a 
split state is entered. This allows the configuration of a split system to be determined while still in die 
combined mode. 

30 Each PCI device is allocated an area of dxe processing set address m^. The top bits of die address are 

determined by die PQ slot Where a device carries out DMA, die bridge is able to check diat die device b 
using die correct address because a D bus arbiter informs die bridge which device is using die bus at a particular 
time. If a device access is a processing set address which is not valid for it, dicn die device access win be 
ignored. It should be noted diat an address presented by a device will be a virtual address which would be 

35 translated by an I/O memory management unit in die processmg set bus controller 50 to an acmal memory 
address. 

The addresses output by die address decoders are passed via die initiator and target controllers 138 and 
140 to the routing matrix 80 via die lines 98 under control of a bridge controller 132 and an arbita 134. 
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An azbher 134 is opr ' ^ m various differcQt modes to arbit"^ for vs- f die bridge on a first-come- . 
first-served basis using conventiona] PQ bus signab on the P -^ad D bases. 

In a combined mode, the arbiter 134 is operable to arbitrate betwer c die in-sync processing sets 14 and 
16 and any initiators on the device bus 22 for use of the bridge 12. PossS> e scenarios are: 
5 - processing set access to the device bus 22; 

- processing set access to internal registos in die bridge 12; 

- Device access to ttkc processing set memory 56. ' 
In split mode, bodi processing sets 14 and 16 must arbitrate the use of die bridge and thtis access to the 

device bus 22 and internal bridge registers (e.g. CSR leginers 114). Hie bridge 12 must also contend with 
10 initiatoR on the device bus 22 for use of diat device bus 22. 

Each slot on the device bus has an arbitration enable bit associated witb it These arbitration enable 
bits are cleared after reset and must be set to allow a slot to request a bus. When a device on the device bus 22 
is 5uq>ected of providing an I/O error, die arbitration enable bit for diat device is automatically reset by the 
bridge. 

15 A PCI bus inter&ce in the processing set bus controller(s) 50 expects to be the master bus controller 

for the P bus concerned, that is it contains die PCI bus arbiter for die PA or PB bus to vMdh it is connected. 
The bridge 12 cannot directly control access to the PA and PB buses 24 and 26. Tbe brieve 12 competes for 
access to the PA or PB bus with the processing set on the bus concerned under ibe control of die bus controUer 
50 on the bus concerned. 

20 Also shown in Figure 8 is a comparatOT 130 and a bridge controller 132. The conq>arator 130 is 

c^>erable to compare I/O cycles from die processing sets 14 and 16 to determine any out-of-sync events. On 
determining an out-of-sync event, the conqiarator 130 b operable to cause the bridge controller 132 to activate 
an Estate for analysis of die out-of-sync event and possible recovery therefrora 
Figure 9 is a schematic functional overview of the routing matrix 80. 

25 The routing matrix 80 comprises a multiplexer 143 which is responsive to initiator control signals 98 

from die initiator controller 138 of Figure 8 to select one of the PA bus path 94 , PB bus path 96, D bus path 92 
or internal bus padi 100 as the current input to the routing matrix. Separate output buffers 144, 145, 146 and 
1 47 are provided for output to each of the paths 94, 96, 92 and 1 00, with those buffers being selectively enabled 
by signals 99 from the target controller 140 of Figure 8. Between the multiplexer and the buffers 144-147 

30 signals are held in a buffer 149. In the present embodiment diree cycles of data for an I/O cycle will be held in 
the p^line represented by die multiplexer 143, die bufTer 149 and the buffers 144. 

In Figures 6 to 9 a functional descrqition of elements of die bridge has been given. Figure 10 is a 
schematic representation of a physical configuration of die bridge in which the bridge control logic 88, the 
storage control logic 90 and die bridge registers 1 10 are implemented in a first field programmable gate amy 

35 (FPGA) 89, the routing matrix 80 is implemented in furdier FPGAs 80.1 and 802 and the SRAM 126 is 
in^lemented as one or more separate SRAMs addressed by a address control lines 127. The bus intei£ices 82, 
84 and 86 shown in Figure 6 are not separate elements, but are integrated in the FPGAs 80.1, 80.2 and 89. Two 
FPGAs 80. 1 and ZOJ. are used for the upper 32 bits 32*63 of a 64 bit PCI bus and die lower 32 bits 0-3 1 of the 
64 bit PCI bus. It will be appreciated diat a single FPGA could be enqiloyed for the routing matrix 80 where 
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die nccessaiy logic can be acconnnodaied wiflim the device. Indeed, «*ere a FPGA of sufficicai edacity i 
available, die bridge control logic, storage control logic and Ae bridge resistcis could be iscoiporated in f-e 
same FPGA as die routing matrix. Indeed many odicr configurations may be envisaged, and indeed technol >gy 
odier dian FPGAs, for example one or more Appbcatira Specific Integrated Circuits (ASICs) ncy be 
employed. As shown in Figure 10. die FPGAs 89, 80.1 and S02 and die SRAM 126 are connected via internal 

bus paths 85 and path control lines 87. 

Figure 11 is a transition diagram ilhistrating in more detail die various operating iiiodcs of die bridges 

nie bridge operation can be divided into dnee basic modes, namely an error state (EState) mode 150, a split 
state mode 156 and a combined state mode 158. TTie EState mode 150 can be ftndier divided i::!o 2 states. 

After initial resetting on powering up die bridge, or foUowing an out-of sync event, die oridge is in diis 
initial Estate 152. In diis state, afl writes are stored in die posted write buffer 120 and reads from die internal 
bridge registers (e g., die CSR registers 1 1 6) are allowed, and aD odier reads are treated as errors (Le. diey are 
aborted). In diis state, die individual processing sets 14 and 16 perform evaluations for deteimining a restart 
time. Each processing set 14 and 16 wiB determine its own restart timer timing. The timer setting depends on a 
"blame- factor for die transition to die EState. A processing set which determines diat it is likely to have caused 
die error sets a long time for die timer. A processing set which diinks it unlikely to have caused die error sets a 
short time for die timer. The first processing set 14 and 16 which times out, becomes a primary processing set 
Accordingly, Ti*ra diis is determined, die briifee moves (153) to die primary Estate 154. 

When eidier processing set 14/16 has become die primary processing set, dw bridge is dien cqxaating 
in die primary EState 154. This state aUows the primary processing set to write to bridge registos (specifically 
dieSRRsll8). Odier writes are no longer stored in die posted write buffer, but are singly lost Device bus 

reads are still aborted in the primary EState 1S4. 

Once die EState condition is removed, die bridge dicn moves (155) to die split state 156. In die split 
state 156. access to die device bus 22 is controUed by die SRR registers 118 while access to die bridge storage is 
simply ariiitrated. TTie primary status of die processing sets 14 and 16 is ignored. Transition to a combined 
operation is achieved by means of a sync_resct (157). After issue of die sync_reset operation, die bridge is then 
operable in die combined state 158. whereby all read and write accesses on die D bus 22 and die PA and PB 
buses 24 and 26 are aUowed. AU such accesses on die PA and PB buses 24 and 26 are compared in d» 
con^arator 130. Detection of a mismatch between any read and write cycles (widi an exception of specific 
dissimilar data VO cycles) cause a transition 15 1 to die EState 150. Tlie various states described are controUed 

by die bridge cootroUer 132. 

Tlie role of die comparator 130 is to monitor and compare I/O operations on die PA and PB buses m 
die combined state 151 and, in response to a mismatched signal, to notify die bridge controller 132, whereby die 
bridge controller 132 causes die transition 152 to die error state 150. Tlie I/O operations can inchide all VO 
operations initiated by die processing sett, as weU as DMA transfers in respect of DMA initiated by a device on 
die device bus. 

Table 1 below summarizes die various access operations ^ch are aUowed in each of die operational 

ststcs 
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TABLE 1 





DBus-Read 


DBus-Wiite 


Estate 


Master Abort 


Stored in Post Write Buffer 


Primaiy Estate 


Master Abort 


Lost 


Split 


Contiolied by SRR bits 


Controlled by SRR bits 




2ad aibitiated 


and arbitrated 


Combined 


Allowed and conqared 


Allowed and compared 



As described above» after an initial reset, the system is in the initial EState 152. In this state, neither 
5 processing sets 14 or 16 can access the D bus 22 or the P bus 26 or 24 of the other processing set 16 or 14. The 
internal bridge registers 1 16 of the bridge are accessible, but are read only. 

A system running in die combined rrx>de 158 transitions to the EState 150 where there is a conq>arison 
failure detected in diis bridge, or ahematively a comparison feihire is detected in another bridge in a muhi- 
bridge system as shown, for example, in Figure 2. Also* transitions to an EState 150 can occur in other 
1 0 situations, for exan^le in die case of a software controlled event forming pan of a self test operation. 

On moving to die EState 150, an interrupt is signaled to all or a subset of the processors of the 
processing sets via an interrupt line 95. Following this, all I/O cycles generated on a P bus 24 or 26 result in 
reads being returned widi an exception and writes being recorded in the posted write buffer. 

The operation of the conq>arator 130 will now be described in more detail. The comparator is 
15 connected to paths 94, 95, 96 and 97 for conqiaring address, data and selected controi ».gnals from the PA and 
PB bus interfaces 84 and 86. A failed corx^arison of in-sync accesses to device I/O bus 22 devices causes a 
move from the conobined state 158 to the EState 150. 

For processing set I/O read cycles, the address, command, address parity, byte enables and pari^ error 
parameters are compared. 

20 If the corxqsarison fails during the address phase, the bridge asserts a retry to the processing set bus 

controllers 50, which prevents data leaving the I/O bus controllers 50. No activity occurs in this case on the 

device I/O bus 22. On the process<»(s) retrying, no error is returned. 

If the comparison fails during a data phase (only control sigiuls ainl byte enables are checked), the 

bridge signals a target-abort to die processing set bus controllers 50. An error is returned to the processors. 
25 In the case of processing set I/O bus write cycles, the address, command, parity, byte enables and data 

parameters are compared. 

If the conqiarison fails during the address phase, the liridge asserts a retry to the processing set bus 
controllers 50, which results in the processing set bus controllers 50 retrying the cycle again. The posted write 
buffer 122 is then active. No activity occurs on the device I/O bus 22. 
30 If the comparison fails during die data phase of a write of)cration, no data is passed to the D bus 22. 

The £uling data and any odier transfer attributes from bodi processing sets 14 and 16 are stored in the 
disconnect registen 1 22, and any subsequent posted write cycles are recorded in the posted write buffer 1 1 8. 
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In case of dirert vttnal mcmcny access (DVm) nads. me data lOT^ 
each dannn. If the data does not natch, the bridge 12 tenmnates the uansfex onthePbus. Inthecaseof 
DVMA writes, control and r-arity eiror signals are checked for correctness. 

Other rignaU in addition to those specifically rnentioncd above can be ccii^^ 

of divergence of the processing sets. Examples of these are bus grants and various specific signals during 

processing set transfers and during DMA transfos. 

Errors M roughly iim) two types, those which are made visible to the software by the pio^ 

bus controller 50 and those which are not made visible by die processing set bus controller 50 and hence need to 
be made visibfe by an interrupt fiom the bridge 12. Accordingly, the bridge is operable to capture errors 
reported in connection widi processing set read and write cycles, and DMA reads and writes. 

Qock control for the bridge is performed by the bridge controller 132 in response to die clock signals 
from die clock line 21. Individual control lines from die controUer 132 to the various elements of the bridge are 

not shown in Figures 6 to 10. . 

Figure 12 is a flow diagram ilhistrating a possible sequence of operating stages where lockstep etrois 

are detected during a combined mode of operanoa 

Stage SI represents dte combined mode of operation where lockstep error checking is performed by 

the comparator 130 shown in Figure 8. 

In Stage S2, a lockstep error is assumed to have been detected by die comparator 130. 

hi Stage S3, die current state is saved in die CSR registers 114 and posted writes are saved in die 
) posted write buffer 122 and/or in die disconnect registers 120. 

Figure 13 ilhistrates Stage S3 in more detail Accordingly, in Stage S31, the bridge controUer 132 
detects whedierdie lockstep error notified by die comparator 130 has occurred during a data phase in which it is 
possible to pass data to die device bus 22. hi diis case, in Stage S32. die bus cycle is terminated. Tben, in Stage 
S33 die data phases are stored m die disconnect registers 120 and control dien passes to Stage S35 where an 
5 evahiation is made as to whedier a fimher VO cycle needs to be stored. Alternatively, if at Stage S31. it is 
determined diat die lockstep error did not occur during a data phase, die address and data phases for any posted 
write I/O cycles are stored in die posted write buffer 122. At Stage S34. if diere are any finther posted write I/O 
operations pending, diese are also stored in die posted write buffer 122. 

Stage S3 is performed at die initiation of die mitial error state 152 shown in Figure 11. In diis state, die 
30 first and second processing sets arbitrate for access to die bridge. Accordingly, in Stage S31-S35, die posted 
write address and data phases for each of die processing sets 14 and 16 are stored m separate portions of die 
postal write buffer 122, and/or in die single set of disconnect registers as described above. 

Figure 14 ilhistrates die source of die posted write l/0=cycles which need to be stored in die posted 
write buffer 122. During normal operation of die processing sets 14 and 16, output buffen 162 in die individual 
35 processors contain lA) cycles which have been posted for transfer via die imiccssing set bus c^^ 

die bridge 12 and eventually to die device bus 22. Simibrly. buffos 160 in die processing set controllers 50 
ah» contain posted I«> cycles for tramfer over die buses 24 and 26 to die bridge 12 and eventually to die 

bus22. 
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Accordingly, it can ^ ^ seen that wbcn an cr*rf state occurs, I/O wri*' ^rycks may already have been 
posted by the p xoc mora 52, cither m their own buffets ' 62, or aireacfy transicncd to the buffeis 160 of the 
processing set bus controllers 50* It is the IA> write cycles in the boifers 162 and 160 which gradually 
pn^gate through and need to be stored in the posted w.ne bufier 122. 
5 As shown in Figure 15, a write cycle 164 posted to die posted write buffo 122 can con^prise an 

address field 165 including an address and an address type, and between one and 16 data fields 166 including a 
byte enable field and the data itself. ^ 

The data is written into iht posted write buffer 122 in the EState unless d)e initiating processing set has 
been designated as a primary CPU seL At that time, non-primary writes in an EState still go to the posted write 
10 buffer even after one of the CPU sets has. become a primary processing set An address pointer in die CSR 
registers 1 14 points to the next available posted write buffer address, and also provides an overflow bit which is 
set w^icn the bridge attempts to write past of the top of the posted write buffer for any one of the processing sets 
14 and 16. Indeed, in the present implementation, only the first 16 K of data is reconled in each buffer. 
Attempts to write beyond the top of the posted write buffer are ignored. The vahie of die posted write buffer 
1 5 pointer can be cleared at reset, or by sofhvare using a write imder the control of a primary processing set 

Retaining to Figure 12, after saving the status and posted writes, at Stage S4 the individual processing 
sets independently seek to evaluate the error state and to determine whether one of the p ro ce s s ing sets is faulty. 
This determination is made by the individual proce ss ors in an er r or state in which diey individually read status 
from the cotttrol state azul EState registers 1 14. During dus error noode, the arbiter 134 arbitrates for access to 
20 die bridge 12. 

In Stage S5, one of the processing sets 14 aiui 16 establishes itself as die primary processing set This 
is determined by each of the processing sets identifying a time factor based on the estimated degree of 
responsibility for the eiror, whereby the first processing set to time out becomes the primary processing set In 
Stage S5, the status is recovered for that processing set and is copied to the other processing set The primary 
^ 25 processing is able to access the posted write buffer 122 and the disconnect registers 120. 

In Stage S6, the bridge is operable in a split mode. If it is possible to re-establish an equivalent status 
for the first and second processii^ sets, then a reset is issued at Stage S7 to put the processing sets in die 
combined mode at Stage SI. However, it may not be possible to re-establish an equivalent state imtil a faulty 
processing set is replaced. Accordingly the system will stay in the Split mode of Stage S6 in order to continued 
30 operation based on a single processing set After replacing the faulty processing set the system could then 
establish an equivalent state and move via Stage S7 to Stage SI . 

As described above, die corxq)arator 130 is operable in the coxnbined mode to coo^iare the I/O 
operations output by the first and second processing sets 14 and 16. This is fine as long as all of die I/O 
operations of the first and second processing sets 14 and 16 are fully synchronized and deterministic Any 
\ 35 deviation from this will be iuteipie ted by the comparator 130 as a loss of lockstep. This is in principle correct 
as even a minor deviation from identical outputs, if not trapped by the comparator 130, could lead to the 
processing sets diverging further from each other as the individual processing sets act on die deviating outputs. 
However, a strict application of this puts significant constraints on the design of the individual processing sets. 
An example of this is that it would not be possible to have independent time of day clocks in the individual 
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processiBg sets operating under their own clocks. This is because ii is impossihlc to obtain t-^-o crystals wbicb 
art 100% identical in operation. Even small differences in die phase of the clocks coiLd be critical as to 
wbedio'^ same san^le is taken at any one time, for example eidter side of a clock transit* on for the respective 
processing sets. 

Accordingiy, a solution to this problem employs the dissimilar data registers (DDR) 1 16 mentioned 
cariier. The sohition is to write data from tiw processing sets into Tt^pcctive DDRs in the te 
die conqsarison of die data phases of the write operations and then to read a selected one of the DDRs backiD 
cadi pnxxssing set, whcrrf)y each of the processing sets is able to act on die same data. 

Figure 17 is a schematic representation of details of the bridge of Figures 6 to 10. It will be noted diat 
details of die bridge not shown m Figure 6 to 8 arc shown in Figure 17, wharas otficr details of die bridge 
shown in Figures 6 to 8 are not shown in Figure 17, for reasons of clarity. 

The DDRs 116 are provided in die bridge registers 110 of Figure 7, but could be provided else^ert in 
die bridge in odicr embodiments. One DDR 116 is provided for each processing set In die cxaii5>le of die 
rauhi-processor system of Figure 1 where two processing sets 14 and 16 arc provided, two DDRs 116A and 
116B are provided, one for each of die first and second processing sets 14 and 16, respectively. 

Figure 17 represents a dissimilar data write stage. The addressing logic 136 is shown schematically to 
comprise two decoder sections, one decoder section 136A for die first processing set and one decoder section 
136B for die second processing set 16. During an address phase of a dissimilar data I/O write operation each of 
die processing sets 14 and 16 outputs the same prcdetcnnincd address DDR-W which is separately interpreted 
by die respective first and second decoding sections 136A and 136B as addressing the respective first and 
second respective DDRs 116Aandll6B. As die same address is output by die first and second processing sets 
14 and 16, diis is not interpreted by the comparator 130 as a lockstep error. 

The decoding section 136A, or die decoding section 136B, or bodi arc arranged to furdicr output a 
disable signal 137 m response to die predetermined write address supplied by the first and second processing 
sets 14 and 16. This disable signal is supplied to die comparator 130 and is operative during die data phase of 
die write operation to disable die con^araior. As a result, die data ou^iut by the first processing set can be 
stored m die first DDR 1 1 6A and die data output by die second processing set can be stored in die second DDR 
116B widiout die conqjarator being operative to detect a difference, even if die data ftom die first and second 
processing sets is different The first decoding section is operable to cause die routing matrix to store die data 
fioin die first processing set 1 4 m die first DDR 1 1 6A and die second decoding section is operable to cause die 
routing matrix to store die data ftom the second processing set 16 in die second DDR 116B. At die end of die 
data phase die comparator 130 is once again enabled to detect any differences between I/O address and/or data 
phases as indicative of a lockstep error. ' 

Following die writing of die dissimilar data to die first and second DDRs 116A and 116B, die 
processing sets are then operable to read the data fiom a selected one of the DDRs 1 1 6A/1 1 6B. 

Figure 18 illustrates an alternative arrangement where the disable signal 137 is negated and is used to 
control a gate 131 at the ou^t of die conqwrator 130. When die dis^le signal is active die output of die 
comparator is disabled, whereas when die disable signal is inactive die output of die conqarator is enabled. 
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Figure 19 iDustiates reading of the first DDR 116A in a subscqce- Hssizmlar cbta read stage. As, 
Hhistrated m Figure 19, each of the processing sets 14 and 16 outputs the same picdctemmed address DDR-^A 
which b separately iuleipi e ted by the respective first and second demding sections 136A and 136B as 
addressing the same DDR, namely die first DDR 116A. As a result, die content of die first DDR 1 16A is read 
5 by bodi of die pr oce ssin g sets 14 and 16, thereby enabling those processing sets lo receive die same data. This 
enables die two processing sets 14 and 16 to achieve deterministic behavior, even if die source of the data 
written mto die DDRs 116 by the pr o cess ing sets 14 and 16 wasnotdetemmcstic. " 

As an ahemative, die pr oces si n g sets could each read the data from the second DDR 1 1 6B. Figure20 
illustrates the reading of die second DDR 1 16B in a dissimilar data read stage foHowing the dissinJlar data 
10 write stage of Figure 15. As iDustrated in Figure 20, each of the processing sets 14 and 16 outputs die & me 
predetermined address DDR*RB which is separately interpreted by the respective first and second decoding 
sections 136A and 136B as addressing the same DDR, namely the second DDR 116B. As a result, the content 
of the second DDR 116B is read by bodi of die processing sets 14 and 16, thereby enabling diose i^cessing 
sets to receive die same data. As widi the dissimilar data read stage of Figure 16, this enables the two 
15 processing sets 14 and 16 to achieve deterministic behavior, even if the source of the data written into the DDRs 
116 by die processing sets Hand 16 was not deterministic. 

The selection of which of the first and second DDRs 116A and 116B to be read can be determined in 
any appiopri ate manner by die software operating on die processing modules. This could be done on die basis 
of a sinq>le selection of one or the other DDRs, or on a statistical basis or randomly or in any other manner as 
20 long as the same choice of DDR is made by bodi or all of die processing sets. 

Figure 21 is a flow diagram summarizing the various stages of operation of the DDR mechanism 
described above. 

hi stage SIO, a DDR write address DDR-W is received and decoded by the address decoders sections 
136A and 136B during the address phase of the DDR write operatioTL 
25 In stage SI 1, die comparator 130 is disabled. 

In stage S12, the data received fiom the processing sets 14 and 16 during the data phase of the DDR 
write operation is stored in the first and second DDRs 1 16A and 1 16B, respectively, as selected by the first and 
second decode sections 136A and 136B, respectively. 

In stage S13, a DDR read address is received from the first and second processing sets and is decoded 
30 by die decode sections 136A and 136B, respectively. 

If die received address DDR-RA is for die first DDR 1 16A, dien in stage S14 die content of diat DDR 
116A is read by bodi ofdie processing sets Hand 16. 

Ahematively, 1 16A if the received address DDR-RB is for die second DDR 1 16B, then in stage S15 
the content of that DDR 1 16B is read by both of the processing sets 14 and 16. 
35 Figure 22 b a schematic representation of the arbitration performed on die respective buses 22, 24 and 

26, and die arbitration for the bridge itself. 

Eaich of the p r o cessin g set bus controllers 50 in the respective processing sets 14 and 16 includes a 
conventional PCI master bus arbiter 180 for providing arbitration to the respective buses 24 and 26. Each of die 
master arbiten 180 is responsive to request signab from the associated processing set bus controller 50 and the 

16 
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bridge 12 on lespecdve request (R£Q) liaes 181 and 182. Iht roaster aibitcrs 180 allocate access to the bos on 
a fiist-come-fiist-scrved basis, issuing a grant (GNT) signal to the winning party on an appropriate grants line 
183 or 184. 

A conventional PQ bus arbiter 1 85 provides axbitration on the D bus 22. The D bus arbiter 185 can be 
5 configured as part of the D bus uxterface 82 of Figure 6 or could be separate therefronL As with the P bos 
master arbrten 186, the D bus arbiter is responsive to request signals from the contending devices, including the 
bridge and ±c devices 30, 31, etc. connected to the device bus 22. Respective request lines 186, 187, 188, etc: 
for each of die entities competing for access to the D bus 22 arc provided for the request sigxials (REQ). The D 
Jbas arbiter 185 allocates access to the D bus on a first-come-first-scrved basis, issuing a grant (GNT) signal to 
10 thcwirmingentity via respective grant lines 189, 190, 192, etc. 

Figure 23 is a state diagram summarising the operation of &e D bus arbiter 185. In a particular 
embodiment iq) to six request signals may be produced by respective D bus devices and one by the bridge itself. 
On a transition into the GRANT state, these arc sorted by a priority encoder and a request signal (REQ#) with 
the highest priority is registered as the winner and gets a grant (GNT#) signal Each winner which is selected 
15 modifies the priorities in a priority encoder so that given the same REQ# signals on ihc next move to grant A 
different device has the highest priority, hence each device has a Tair^ chance of accessing DEVs. The bridge 
R£Q# has a highCT weighting than D bus devices and will, under very busy conditions, get the bus for every 
second device. 

If a device requesting the bus faik to perform a transaction within 16 cycles it may lose GNT# via die 
20 BACKOFF state. BACKOFF is required as, uiider PQ rules, a device nuy access the bus one cycle after GNT# 
is removed. Devices may only be granted access to D bos if the bridge is not in die not in the EState. A riew 
GNT# is produced at the tinoes when the bus is idle. 

In the GRANT and BUSY states, the FETs are enabled and an accessing device is known and 
forwarded to die D bus address decode logic for checking against a DMA address provided by die device. 
25 Turning now to die bridge arbiter 134, this allows access to die bridge for the first device which asserts 

die pa FRAME# signal indicating an address phase. Figure 24 is a state diagram summarising die operation of 
die bridge arbiter 134. 

As with the D bus arbiter, a priority encoder can be provided to resolve access atteirqits which collide. 
In this case "a collision" the loser/losers are retried which forces diem to give up the bus. Under PQ ndes 
30 retried devices must try repeatedly to access die bridge and this can be expected to h^ipen. 

To prevent devices which arc very quick with their retry attempt from hogging the bridge, retned 
interfaces are remembered and assigned a higher priority. These remembered retries are prioritised in the same 
way as address phases. However as a precaution this mechanism is timed out so as not to get stuck waiting for a 
faulty or dead device. The algorithm employed prevents a device which hasn't yet been retried, but which 
V 35 woiddbeahigherpriority retry than a device currently waiting fOT, from being retried at die first a 

In combined operations a PA or PB bus iiqmt selects which P bus interface wxD win a bridge access. 
Bodi are informed diey won. Allowed selection enables latent fauh checking durii^ normal operation. EState 
prevents die D bus from winning. 
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Thetxridgeaxbiter ir ic^MOST'c u> stazulard PQ signals provided tasdaid PQ contToI lines 22,' 
24 and 25 to control acce';. to die bridge 1 i. 

Figure 25 illustrates signals associated with an I/O operation cycle on the Pd bos. A PQ frame signal 
(FRAM£#) is initially asserted. At die same time, address (A) signals wiH be available on the DATA BUS and 
5 tbe appropriate command (write/read) signals (C) wiU be available on the m i i mm i K j bus (CMD BUS). Shortly 
after die frame dgnal being asserted low. the initiator ready signal (IRDY#)wiO also be ass Whenthe 
device re^^onds, a device selected signal (DEVS£L#) will be asserted low. When a target ready signal is- 
asserted low flUD Y#), data transfer (D) can occur on die data bus. 

The bridge is opctable to allocate access to the bridge resources and thereby to negotiate allocation of a 
10 target bus in re^nse to tL ; FRAM£# being asserted low for die initiator bus cancemed. Accordingly, die 
bridge arbiter 134 is operable to allocate a c cess to the bridge resources and/or to a target bus on a Grst-come- 
flrst-served basis in response to die FRAMED being asserted bw. As well as die single first-conv-fxrst-served 
basis, the aibiters may be additionally provided with a mechanism for logging die arbitration requests, and can 
in^ly a conflict resohidon based on tbe request and allocation history v/bext two requests are received at an 
15 identical time. Alternatively, a simple priority can be allocated to the various requesters, whereby, in die case 
of identically timed requests, a pardcular requester always wins the aUocation process. 

Each of die slots on the device bus 22 has a slot response register (SRR) 118, as weU as other devices 
connected to die bus, such as a SCSI interface. Each of the SRRs 118 contains bits defining die ownership of 
the slots, or die devices connected to the slots on the direct memory access bus. In thb embodiment, and for 
20 reasons to be elaborated below, each SRR 1 18 comprises a four bit register. However, it will be appreciated 
diat a larger register will be required to determine ownership between more dian two processing sets. For 
exanqile, if diree processing sets are provided, then a five bit register will be required for each slot 

Figure 16 illustrates schematically one such four bit register 600. As shown in Figure 16, a first bit 
602 is identified as SRR[0], a second bit 604 is identified as SRR[ 1 ], a diird bit 606 is identified as SRR[2] and 
25 a fourth bit 608 is identified as SRR{3]. 

Bit SRR[0] is a bit which is set when writes for valid transactions are to be suppressed. 

Bit SRR[1] is set when the device slot is owned by the first processing set 14. This defines the access 
route between the first processing set 14 and the device slot When this bit is set, the first processing set 14 can 
ahvays be master of a device slot 22, while the ability for die device slot to be master depends on whether bit 
30 SRR[3]i5seL 

Bit SRR[2] is set when the device slot is owned by the second processing set 16. This defines die 
access route between die second processing set 16 and the device slot When diis bit is set, the second 
processing set 16 can always be master of die device slot or bus 22, while the ability for die device slot to be 
master depends on whether bit SRR[3] is set 
35 Bit SRR{3] is an arbitration bit which gives the device slot the ability to become master of die device 

bus 22, but only if it is owned by one of die processing sets 14 and 16, tiiat is if one of die SRR [1) and SRR[2] 
bits is set 

When die fake bit (SRR[0]) of an SRR 118 is set, writes to die device for diat slot are ignored and do 
not appear on the device bus 22. Reads return indeterminate data without gaita^g a transaction on die device 

18 
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bos 22. In tbc event of an I/O cttot the feke bit SRR[0) of t^e SRR 188 tre^Hmding to the device wbich 
caused die error is srt by tfic hardware configuratioD of die bridge to disai ';c furticr access to die device slot 
concerned. An inicrri^ inay also be generated by die bridge to inform die software ^ 
leading to die I/O errw diat die error has occurred. The fake bit has an -ffect «Acdicr die system is in die split 
5 (ff the combined mode of operation. 

The ownershq) bhs only have effect, however, in die split system mode of operation. In diis mode, 
each slot can be in dirte states: ~ 
Not-owned; 

Owned by processing set 14; and 
10 Owned by processing set 16 

TTiis is determined by die two SRR bits SRRIl] and SRR{21. widi SRRII] being set when die slot is 
owned by processing set 14 and SRRI2] being set when die slot is owned by process If die slot is un- 

owned, dicn neidier bit is set (bodi bits set is an illegal condition and is prevented by die hardware). 

A slot whidi is not owned by the processing set making die access (diis inchides un-owned slots) 
15 cannot be accessed and results in an abort A processing set can only claim an un-owned slot; it cannot wrest 
ownership away from anodier processing set This can only be done by powcring-off die odicr processing set 
When a processing set is powered off, all slots owned by it move to die un-owned state. Whilst it is not 
possible for a processing set to wrest ownership from anodicr processing set, it is possible for a processing set to 
give owncrsh^ to anodier processing set 
20 The owned bits can be altered when in die combined mode of operation state but dicy have no effect 

untQ the split mode is entered. 

Table 2 below summarizes the access rights as detcraiined by an SRR 11 8. 

From Table 2, it can be seen diat when die 4-bit SRR for a given device is set to 1100, for exan^le, 
dien die slot is owned by processing set B (Le. SRRI2] is logic higji) and processing set A may not read from or 
25 write to die device (i.e. SRR[11 is logic low), aldiough it may read from or write to die bridge. "FAKE_AT" is 
set logic low (i.e. SRRIO] is logic low) indicating diat access to die device bus is allowed as dierc are no faults 
on die bus. As "ARB.EN" is set logic high (i,e. SRKI3] is logic high), die device widi which die register is 
associated can become master of die D bus. This exanq>lc demonstrates die operation of die register when die 
bus and associated devices are operating correctly. 
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TABLE 2 



5 


SRR 

PPiiiHO] 


PABUS 


PBBUS 


Device Inteiface 




0000 
xOOx 


Rcaxl/Writc bridge SRR 


Read/Write bridge SRR 


Access denied 


10 


0010 


Rcad/WritB bridge 
OwncdDSlot 


Read/Write bridge 
No access to D Slot 


Access Denied because 
axbinanon bit is off 




0100 


Read/Write bridge 
No access to D Slot 


Read/write bridge 
Access to D Slot 


Access Denied because 
aibidation bit is ofT 


15 


1010 


Read/Write bridge. 
Owned DSbt 


ReadAVrite Bridge 
No access to D Slot 


Access to CPU B Lenied 
Access to CPU A OK 


20 


1100 


Read/Write bridge^ 
No access to D Slot 


Read/Write bridge 
Access to D Slot 


Access to CPU A Denied 
Access to CPU B OK 


0011 


Read/Write bridge. 
Bridge discard writes 


Read/Write bridge 
No access to D Slot 


Access Denied because 
Aibitration bit is off 


25 


0101 


Read/Write bridge. 
No access to D slot 


Read/Write bridge 
Bridge discards writes 


Access Denied because 
Aibitrarion bit is off 




1011 


Read/Write bridge. 
Bridge discard writes 


Read/Write bridge 
No access to D Slot 


Access to CPU B Denied 
Access to CPU A OK 


30 


1101 


Read/Write bridge. 
No access to D slot 


Read^rite bridge 
Bridge discards writes 


Access to CPU B Denied 
Access to CPU A OK 



In an alternative example, where the SRR for the device is set to 0101, the setting of SRR[2] logic high 

35 indicates that the device is owned by processing set B. However, as the device is malfunctioning, SRR{3] is set 
logic tow and the device is not allowed access to the processing set SRR[0] is set high so that any writes to the 
device are ignored and reads therefrom return indeterminate data. In this way, the maliunctioning device is 
effectively isolated from the processing set, and provides indeterminate data to satisfy any device drivers, for 
exan:q>le, that might be looking for a refuse from the device. 

40 Figure 26 illustrates the operation of the bridge 12 for direct memory access by a device such as one of 

the devices 28, 29, 30, 31 and 32 to the memory 56 of the processing sets 14 and 16. When the D bus arbiter 
185 receives a direct memory access (DMA) request 193 from a device (e.g., device 30 in slot 33) on the device 
bus, the D bus arbiter determines whether to allocate the bus to that slot As a result of this granting procedure, 
the D-bus aibiter knows the slot which has made the DMA request 193. The DMA request is supplied to the 

45 address decoder 142 in the bridge, where the addresses associated with the request are decoded. The address 
decoder is re^nsive to the D bus grant signal 194 for the slot concemed to identify die slot which has been 
granted access to die D bus for the DMA request 

The address decode logic 142 holds or has access to a geographic address map 196, which identifies 
the relationship between the processor address space and the slots as a result of the geographic address 

50 enqployed. This geographic address map 196 could be held as a table in the bridge memory 126, aloi^ with die 

20 
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posted write buffer 122 aiw j« dirty RAM 124. Alternatively, it could he - as a table in a separate metnoiy 
element, possibly forming part of the address decoder 142 ttsclfl Tbc map 1S2 could be configured in a form 
other dian a table. 

The address decode logic 1 42 is configorcd to verify die correctness of the DMA addresses supplied by 
5 the device 30. In one embodiment of ibe invention, diis is achieved by cozc^nxing four significs^ui address bits 
of the address supplied by die device 30 with die corresponding four addicss bits of tbc address held in the 
geogr^Aic addrr^^g map 196 for die slot identified by die D bus grant signal for die DMA request In diis- 
exan^le, four address bits arc sjifficient to determine whether die address sillied is widim die correct address 
range. In diis specific cxan^le, 32 bit PQ bus addresses are used, with bits 31 and 30 always being set to 1, bit 
10 29 being aDocated to identify which of two bridges on a modierboard is bcmg addressed (sec Figure 2) and bits 
28 to 26 identifying a PQ device. Bits 25-0 define an effect from die base address for die address range for 
each slot Accordingly, by conq>aring bits 29-26, it is possible to identify whedier die address(es) supplied 
fall(s) widun die ^jpropriatc address range for die slot concerned. It will be qjprccialed tiiat in odier 
embodiments a different number of bits may need to be compared to make diis determination depending upon 
IS the aUocation of die addresses. 

The address decode logic 142 could be arranged to use die bus grant signal 184 for the slot concerned 
to identify a table entry for die slot concerned and dien to conq>are die address in fliat entry widi die address(es) 
received widi die DMA request as described above. Alternatively, die address decode logic 142 could be 
arranged to use die address(es) received widi die DMA address to address a relational geographic address map 
20 and to determine a slot number dicrefrom, which could be compared to die slot for which die bus grant signal 
194 is intended and thereby to determine whether the addresses fall within the address range appropriate for the 
slot concerned. 

Eitiier way, die address decode logic 142 is arranged to permit DMA to proceed if die DMA addresses 
faU witiiin die expected address space for die slot concerned. Odicrwisc, die address decoder is arranged to 
25 ignore die slots and die physical addresses. 

The address decode logic 142 is further operable to control the routing of die DMA request to the 
appropriate processing set{s) 14/16, If die bridge is m die combined mode, die DMA access will automatically 
be allocated to aU of die m-sync processing sets 14/16. The address decode logic 142 wifl be aware diat die 
bridge is in die combined mode as it is under die control of die bridge controller 132 (sec Figure 8). However, 
30 where die bridge is in die split mode, a decision will need to be made as to which, if any. of die processing sets 
the DMA request is to be sent 

When die system is in split mode, die access will be directed to a processing set 14 or 16 whidi owns 
die slot concerned. If die slot is un-owned, then die bridge does not re^nd to die DMA request In die qplit 
mode, die address decode logic 142 is operable to determine the ownership of the device orighaating the DMA 
35 ropiest by accessing die SRR 118 for die slot concerned. The appropriate slot can be identified by die D bus 
grantsignaL The address decode logic 142 is operable to control the target controller 140 (see Figure 8) to pass 
die DMA request to die appropriate proccssmgset(s) 14/16 based on die ownership bits SRR( 11 and SRRI2]. If 
bit SRR[1] is set die first processing set 14 is die owner and die DMA request is passed to die first processing 
set IfbitSRRP] is set, die second processing set 16 is die owner and die DMA request is passed to die second 
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processiBg set ITndther of' ^ SRR[1] and SRR[2] is set, then the DMA test is igBorcd by the address 
decoder^ is not passer to exdier of the processing sets 14 and 16. 

Hgure 27 is a low diagram summarizing the DMA verification process as illustiated widi reference to 
Figure 24. 

5 In stage S20, die D-bus arbiter 160 arbitrates for access to the D bus 22. 

In stage S2U the address decoder 142 verifies die DMA addresses supplied widi die DMA request by 
aecessing die geogi^ihic address m^. - 
in sUge S22, die address decoder ignores die DMA access where the address faOs outside the expected 
range for die slot concerned. 

10 AHematively, as rqiresented by stage S23, the actions of the address decoder are dependent upon 

whedier die bridge is in die combined or the split mode. 

If the bridge is in the combined mode, then in stage S24 the address decoder controls the target 
controller 140 (see l^gure 8) to cause die routing matrix 80 (see Figure 6) to pass die DMA request to both 
processing sets 14 and 16. 

15 If die bridge is in the split mode, the address decoder is operative to verify the ownership of the slot 

coiiceraed by reference to the SRR 1 1 8 for diat slot in stage S25. 

If die slot b allocated to die first processing set 14 (Le. the SRR[1] bit is setX dien in stage S26 die 

address decoder 142 controk the target controller 140 (see Figure 8) to cause the routing matrix 80 (see Figure 

6) to pass die DMA request to first processing set 14. 
20 If the sk>t is allocated to the second processing set 16 (Le. the SRR[2] bit is set), dien in stage S27 die 

address decoder 142 controls die target controller 140 (see Figure 8) to cause die routing matrix 80 (see Hgure 

6) to pass the DMA request to the second processing set 16. 

If die slot is unallocated (Le. neither the SRR[1] bit nor the SRR[2] bit is set), then in step SI 8 the 

address decoder 142 ignores or discards the DMA request and the DMA request is not passed to the processing 
25 sets 14 and 16. 

A DMA, or direct vector memory access (DVMA), request sent to one or more of the processing sets 
causes the necessary memory operations (read or write as appropriate) to be effected on the processing set 
memory. 

There now follows a description of an example of a mechanism for enabling automatic recovery from 

30 an Estate (see Figure 11). 

The automatic recovery process inchuks reintegration of the state of the pr oce ssin g sets to a common 
status in order to atten^t a restart in lockstqi. To achieve diis, the pnicessiiig set in^iidi asserts itself as die primary 
processing set as described above copies its con^lete state to die odier processing This involves ensuring diat 
die content of the memory of bodi processors is die same before trying a restart in lockstep mode. 

35 However, a problem widi the copying of die content of die memory from one p rv^^ing set to die odicr is 

that during diis copyii^ process a device con ne c te d to the D bus 22 might attempt to make a direct memory access 
(DMA) request for access to the memory of die primary processing set If DMA is enabled, dien a write made to 
an area of niemoiy which has already been cc^iied would result in die tneniory state of the ^ 
of die copy not being die same. In principle, it would be possible to inhibit DMA for the whole of die cajpy 
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process. However, flus would be undesirable bearing in mb d that ii is dcsirabk to mininnse ihc time tbat die 
system or die resources of die system aie unavaiiabk. As a>j ahemativc, it wuld be possible to retry die ^le 
cqjy operation wbcn a DMA operation has ocnmed duri jg die period of the copy. However, it is likely diat 
fardicr DMA operations would be performed during dw copy retry, and accordingly diis is not a good option 
eidier, Acccndingly. in die prescm sysaai, a dirty RAM 124 is prov^ As described carKer die 

dirty RAM 124 is configured as part of die bridge SRAM menwry 126. 

The dirty RAM 124 con^irises a bit map hsving a dirty indicatOT^ 
or page, of memory. The bit for a page of memory fe set when a write access to die area of rncmory conce^ 
made. In an embodimcm of die invcntitmcmc bit is provided fw every 8K page 

The bit for a page of processing set memory is set automatically by die address decoder 142 when diis decodes a 
DMA request for diat page of memory for cidicr of die processing sets 14 or 16 from a device connected to die D 
bus22. TTk dirty RAM can be reset, or cleared when it is read by a processing set. fo^ 
and clear mstructions at die beghming of a copy pass, so diat it can ^ 
gtventime. 

The dirty RAM 124 can be read word by wOTd. If a large word size is chosen for reading die dirty RAM 
124, this will optimise the reading and resetting of the dirty RAM 124. 

Accordingly, at die end of die copy pass die bits in die dirty RAM 124 wifl indicate diose pages of 
jOTcessing set memory have been changed (or di^ ^ 
fuidier copy pass can dien be performed for only diose pages o Thiswilltake 
less time diat a fan copy of die memory. Accordingly, dicrc are typically less pages rnariced as dirty at da 
die next copy pass and, as a resuh, die copy passes can become shorter and shorter. As some time it is necessary to 
decide to inhibit DMA writes for a short period fora final, short, copy pass, at die end of i»*diich die memories of 
die two processing sets will be die same and die primary processing set can issue a reset operation to restart die 
combined mode. 

The dirty RAM 124 is set and cleared in bodi die combined and split modes. This means diat in split 
mode die dirty RAM 124 may be cleared by eidier processing set. 

The dirty RAM 1 24 address is decoded from bits 1 3 to 28 of die PQ address presented by die D bus 
device. Erroneous accesses which present illegal combinations of die address bits 29 to 31 are mapped into die 
dirty RAM 124 and a bit is dirtied on a write, even diough die bridge will not pass diesc transactions to die 
processing sets. 

When reading die dirty RAM 1 24, die bridge defines die whole area from 0x00008000 to OxOOOOfHT 
as dirty RAM and will clear the contents of any location in diis range on a read. 

As an alternative to providing a single dirty RAM 124 which is ctered on being read, anodicr 
alternative would be to provide two dirty RAMs which are used in a toggle mode, widi one being written to 

v^iile another is read. 

Figure 28 is a flow diagram summarising die operation of the dirty RAM 124. 

In stage S41. die primary processing set reads die dirty RAM 124 which has die effect of resetting die 
dirty RAM 124. 
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In stige S42, the piii ' ' y processor (eg. proccssmg set 14) copies thr -"boL of its memory 56 to tbc 
memcny 56 of die other piDcesb^ set (e.g. processing set 16). 

Ib stage S43, ±s primaiy processing set reads the dizty RhM 124 ^Kdiich has 6^ effect of resetting the 
dirty RAM 124. 

In stage S44, die piiiuaiy processor determines Miether less than a predstertiined number of bits have 
been written in the diz^ RAM 124. 

If mofc dian &e predetennined Tnrmbcr of bits have been set, dicn the processor in stage S45 copies dios^ 
pages of its memory 56 viiich have been dirtied, as indicated by &e dirty bits read fiom die dixty RAM 124 in 
stage S43, to die mertKny 56 of the odier processing set Control dien^-tfses back to stage S43. 

I£ in stage S44, it is detexnmied less than the pxcdetcmiiiied nuiDbi t of bits have been written in the dirty 
RAM 124, dien in Stage S45 die primary processor causes die bridge to inhibit DMA requests from die devices 
connected to die D bus 22. This could, for cxanqile, be achieved by clearing the arbitration enable bit for rrh of 
die device slots, dierdyy denying access of die DMA devices to die D bus 22. Ahemativety, die address decoder 
142 could be configured to ignore DMA requests under instructions from the primary processor. During the period 
in ^^ch DMA accesses are prevented, the primary processor dicn makes a final copy pass from its mcrooiy to the 
iriemory 56 oftfaeodier processor for dioseihenioiy pages corxespondmg to die bx^ 124. 

In stage S47 the primary processor can issue a reset operation for initiating a cooibitted mode. 

In stage S48, DMA accesses are once nxne pcimiQed. 

It win be appreciated diat aldiough paTtictiiar embodiments of die invention have been described, many . 
modifications/additions and/or substitutions may be made widiin the ^lirit and scope of die present invcnticm as 
defined in the appended claims. For example^ although in the specific description two puM je gAu ig sets are provided, 
it win be appreciated that the specifically described features may be modified to provide for duee or inrae 
processii^ sets. 
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1. A bridge fc^ a multi-processor system, die bridge compiising a first processor bus interface for 
comiectioD to an I/O bus of a first processing set, a second processor bus interface for cormection to an I/O bus 

5 of a second processing set. a device bus interface for cormection to a device bus and a bridge control mechanism 
configured to be operable to monitor operation of the first and second processing sets in a combined, lockstcp, 
operating mode and to resporul to detection of a lockstep error to cause the bridge to be operable in an error- 
mode in which write accesses initiated by the processing sets are stored in a bridge buffer pending resohition of 
the error mode. 

10 

2. The bridge of claim 1, wherein a respective buffer region is provided for each processing set 

3. The bridge of claim 1, wherein the bridge control mechanism is configured to be operable in an initial 
error noode: 

15 to store in the posted write buffer any internal bridge write accesses issued by the processing sets and 

to allow and to arbitrate any internal bridge read accesses initiated by the processing sets; and 

to store in a posted write buffer any corr^lete device write accesses initiated by the processing sets and 
to abort any device bus read accesses initiated by the processing sets. 

20 4 The bridge of claim 3, wherein the bridge control mechanism is operable in the initial error noode, 
where data is in transit through.the bridge to divert the data to one or more discormect registers. 

5. The bridge of claim 1, wherein the bridge control mechanism is configured to be operable, in a primary 
error mode in which a processing set asserts itself as a primary processing set 

<25 to allow and to arbitrate any internal bridge write accesses initiated by the primary processing set, to 

discard any internal bridge write accesses initiated by any other processing set, and to allow and to arbitrate any 
internal bridge read accesses initiated by the processing sets; and 

to discard any device bus write accesses initiated by the processing sets and to abort any device bus 
read accesses initiated by the processing sets. 

30 

6. The bridge of claim 1 , comprising a controllable routing matrix coupled between the first processor bus 
interface, the second processor bus interface, the device bus interface and the memory sub-system, the bridge 
control mechanism being configured to be operable to control fheirouting matrix selectively to inteicoimect the 
first processor bus interface, the second processor bus interface, the device bus interface and the memory sub- 

\ 35 system according to a current iriode of operation. 

7. The bridge of claim 6, wherein the bridge control mechanism contprises: 

an address decode mechanism configured to be operable to determine destination addresses for write 
and read accesses; and 
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initiator and tai;get cr ^Deis for detenttimng a path from an ix!pst to output of the routing Twatrix^ 
iiie initia or controller being re^>onsive to die determination of a master for a transfer in progress and the target 
controV^r being re^xKisive to decoded destinaticm addresses from die aduiess dccodg mechaziism. 



5 8. Tbe bridge of claim 7, wherein tiie bridge control mechanism compxiscs a conqyaraior configmcd to be 
operable in the combined mode to detect differences between signals on die I/O buses of tbt fiist and second 
jHOcessing sets as indicative of a lockstep error. — 

9. The bridge of claim 8, wherein the bridge control mechanism further comprises a bridjge controller 
: 0 connected to an output of the coxx^Mffator, die bridge controller being configured to be operable, in response to a 

signal indicative of a lockstep error ou^ut from the conq)arator, to cause the bridge to cease operation in the 
combined mode and instead to operate in die error mode. 

10. The bridge of claim 9, wherein the master and target controUers are frirdier responsive to an output of 
15 die bridge controller to modify the padi from the input to the output of die routing matrix. 

1 1 . The bridge of claim 3, wherein die posted write buffer is configured in random access memory. 

12. The bridge of claim 4, wherein the at least one disconnect register is a hardware register. 

20 

13. The bridge of claim 1, comprising at least one further processor bus interface for connecdon to an I/O 
bus of a further processing set 

14. A bridge for a multi-processor system, the bridge con^nising means fot interfacing with an I/O bus for 
25 each of first and second processing sets and with a device bus, a posted write buffer, means for monitoring 

operation of the first and second processing sets in a lockstep mode and means responsive to detection of a 
lockstep error to store write operations posted by die processing sets in the posted write buffer. 

15. A computer system con^msing a first processing set having an I/O bus, a second processing set having 
30 an I/O bus, a device bus and a bridge* the bridge being connected to the I/O bus of the first processing set, the 

I/O bus of the second processing set and die device bus and conqmsing a bridge control mechanism configured 
to be operable to monitor (^>eration of die first and second processing sets in a combined, lockstep, operating 
mode and to respond to detection of a lockstep error to cause t^e bridge to be operable in an error mode in 
which write accesses initiated by die processing sets are stored in a bridge buffer pending resolution of the error 
35 mode. 

16. A computer system according to claim 15, wherein each processing set comprises at least one 
processor, memory and a processing set I/O Ims controller. 
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17. The cowptiter jn of ciaim 15, f* rther couqmsixig at least asc i. jgr p ru ccssing set 

18. A method of operating a multi-p: jcessor system comprising a first processing set having an I/O bus, a 
second processing set having an I/O bus a device bus and a bridge, the bridge being cozmected to the I/O bus of 

5 the first processing set, die I/O bus of the second processing set and the device bos and conqirising a bridge 
buffer, the method conqirising; 

monitoring operation of the first and second processing sets in a combined, lockstep, operating mode; 

and 

on detection of a iockstep error, storing write accesses initiated by the processing sets in the bridge 
10 pending resolution of the enor. 

19. The method of claim 18, wherein write accesses for each processing set are buffered in respective 
buffer regions for each processing set 

15 20. The method of claim 18, wherein, in an initial error nuxie: 

any internal bridge write accesses issued by the processing sets are buffered in the posted write buffer 
and any internal bridge read accesses initiated by the processing sets are allowed and arbitrated; and 

any con^lete device write accesses initiated by the processing sets are stored in a posted write buffer 
and any device bus read accesses initiated by die processing sets are aborted. 

20 

21. The method of claim 20, wherein: 

in the initial error mode, where data is in transit through the bridge, the data is diverted to one or more 
disconnect registers. 

25 22. The method of claim 1 8, wherein, in a primary error mode in which a processing set asserts itself as a 
primary processing set 

any internal bridge write accesses initiated by the primary processing set are allowed and arbitrated, 
any internal bridge write accesses initiated by any other processing set are discarded, and any internal bridge 
read accesses initiated by the processing sets are allowed and arbitrated; and 
30 any device bus write accesses initiated by the prt>cessing sets are disca r ded and any device bus read 

accesses initiated by the processing sets are discarrird 

23. The method of claim 1 7, wherein a controllable routing ^trix is connected between the I/O buses, die 
data buses and bridge memory, the routing matrix being controlled selectively to intercoimect the first processor 
35 bus interface, the second processor bus interface, the device bus inter&ce and tibe bridge memory according to a 
current mode of operation. 
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24. The method of daxm 22, ^t^wiein: 

an address decode l aiasm conoected to Ac fjzu pr^cessM bos h .ace. fee second proccsscr Inis' 
interface and die device bos mtex£ice« and is opexabie to detenniss ^nrce and destination addresses for write 
and read accesses; and 

initiator and target cantroUcis for determining a path fir m an input to an oa^put of tbe routing matrix 
are responsive to identification of a master and to decoded destination addresses* respectively. 

25. Themeduxlofdaim23,^»dkexc]n: 

a conqsarator connected to die first and second processor bus intcr&ccs is operable in the combined 
mode to d et e ct differences between signals on the I/C buses of the first apd second process ing sets as indicative 
of a lodcstep error. 

26. Tbemethodof claim 24, wherein: 

a bridge coutroOer connected to an output of the con^iarator is operable, in response to a signal 
indicative of a lodcstep error ou^ut fiom the comparator, to cause the bridge to cease operation in the combined 
xnode and instead to operate in the enor mode. 

27. The method of claim 25, v^ierein die master and target controllers are fimher responsive to an output 
of dte bridge controller to modify the path from the input to the output of the routing matrix. 
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